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10 TECHNICAL FIELD 

The present invention concerns a metlnoci of indexing a digital 
image taking into account the spatial distribution of the visual content of the 
image. 

The invention also concerns a method of searching for images 
15 from an example image, amongst a plurality of images stored in a database, 
the images being indexed according to the above-mentioned indexing method. 

The invention also concerns a device capable of implementing the 
aforementioned image indexing and search methods. 

BACKGROUND OF THE INVENTION 
20 A conventional method of searching for digital images in a 

database containing images is based on a system of indexing images in the 
database. 

The aim of an indexing system is to associate, with each image in 
the database, an information item characteristic of the content of the image 
25 referred to as the "image index". The set of these information items constitutes 
what is designated by the "database index". Typically, the content of an image 
is characterised by its colour distribution (histogram), its texture or its form. 

A user interrogates a database of images through a query 
containing an information item characteristic of the image type sought. 
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Conventionally, this query is constituted by an example image. The user 
chooses an image, from the database or external to it, whose content is similar 
to the image type sought. The content of the example image is then compared 
with the content of the database index according to a search strategy. Finally, 
5 the image In the database whose indexed information item has the greatest 
similarity to the content of the example image is extracted. Generally, a number 
of images are extracted from the database to be presented to the user, 
arranged according to the significance of their similarity with the query. The said 
user then makes a choice from amongst the images presented. 

10 Conventionally, there are two approaches for defining the search 

strategy. It may be of the linear type or the hierarchical type. 

The linear approach consists of considering the database as a 
single vector, each component of which is an image associated with an Index. 
The strategy then consists of calculating the similarity between the index of the 

1 5 example image and that associated with each of the components of this vector. 
This strategy is easy to implement, which explains why it is often used. 
However, when the database contains a large number of images, of the order 
of several thousand, the calculation time for the search may be very long, which 
penalizes accordingly the time for responding to a query. 

20 In order to reduce the above-mentioned calculation time, the 

hierarchical approach can be used. One known hierarchical search technique 
uses, for each image in the database, an index composed of two distinct parts 
which will be referred to here as "sub-indices". Amongst these two sub-indices, 
a first sub-index is of a not very discriminant type, while the second sub-index is 

25 of a more discriminant type. 

The search process then takes place in two steps. During a first 
step referred to as a "filtering step", a "coarse" sorting of the images in the 
database is carried out using the first sub-indices. At the end of this step, a 
subset of images is selected. A second step, referred to as a "matching step" is 

30 then carried out, this time performed on the subset of images retained during 
the first step, during which the search is refined using the second sub-indices. 



At the end of this second step, only the images most resembling the example 
image are kept. 

An example of a hierarchical search method is given in the article 
entitled "Multiresolution video indexing for subband video database", by J. Lee 
and B. W. Dickinson, Proc. of SPIE: Storage and retrieval for images and 
videos databases, vol. 2185, San Jose, CA, February 1994. The search 
method taught in this article applies to images compressed by a subband 
decomposition technique. Each of the subbands of an image has an index 
associated with it. According to this search method, a start is made by 
comparing the indices associated with the so-called "low frequency" band of the 
images, then the search result obtained is refined by a comparison of the 
indices associated with the higher frequency bands. 

Generally speaking, the known hierarchical search methods of the 
prior art use indices which characterise the overall content of the images. Thus, 
in the search method taught in the above-mentioned document, the indices 
characterise the overall content of the image according to a number of levels of 
resolution. However, in these search methods, the index of a given image never 
characterises the visual content of this image according to its spatial distribution 
over the whole image medium. The taking into account of the spatial 
distribution of the content by the image index would nevertheless make it 
possible to increase the accuracy of this index and therefore the accuracy of 
the search using such an index. 

SUMMARY OF THE INVENTION 

The aim of the present invention is to propose an indexing method 
by which the visual content of a given image is characterised notably by its 
spatial distribution over the whole image medium, and a search method using 
the said indexing method. 

To that end, the present invention concerns, according to a first 
aspect, a method of indexing a digital image, characterised in that it comprises 
the following steps: 

- generating a first information item (H(lm)) characteristic of the 
visual content of the said image (Im); 
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- generating a second information item (W(lm)) characteristic of 
tlie spatial distribution of the visual content of the image (Im) in its image plane; 

- associating, with the said image (Im), an index (IDX(lm)) 
composed of the first information item (H(lm)) and the second information item 

5 (W([m)). 

Thus, an image index obtained by this method represents the 
visual content of the image through the first information item (H(lm)) included in 
it, and also represents the spatial distribution of the content in the image plane 
through the second information item (W(lm)) included in it. Consequently, this 

10 indexing method improves the representativeness of the index with respect to 
the image to which it relates, and increases the accuracy of an image search 
based on the use of such indices. 

According to an advantageous characteristic of the invention, the 
step of generating the first information item (H(lm)) comprises the following 

15 substeps: 

- dividing the image plane of the said image (Im) according to a 
partitioning comprising a predefined number N of blocks (Bi); 

- extracting, from each of the said blocks (Bi), a data item of a first 
type (/z/™) representing at least one characteristic of the visual content of the 

20 block under consideration; 

- generating the said first information item (H(lm)) as being a 
vector having N components, each of which is one of the data items of the first 
type(Af ). 

Thus, the first information item (H(lm)), characteristic of the visual 
25 content of the image, represents the local content (inside the blocks) of the 
image, which also increases the accuracy of the index. 

According to a first embodiment of the invention, the step of 
generating the second information item (W(lm)) comprises the following 
substeps: 

30 - calculating, for each of the blocks (BO, a data Item of a second 

type (w,"") indicative of a degree of significance of the visual content of the 



block (Bi) under consideration with respect to the overall content of the image 
(Im); 

- generating the second information item (W(lm)) as being a 
vector having N components, each of which is one of the data items of the 
second type ( wj"" ). 

By measuring the significance, for each of the blocks under 
consideration of the image, of the local content of the block with respect to the 
overall content of the image, there is finally obtained a second information item 
(W(lm)) which Is indicative of the spatial distribution In the image plane of the 
first information item (H(im)), characteristic of the visual content of the image. 
The second information item thus calculated makes it possible to make the 
image index effectively represent the spatial distribution of the visual content of 
the image In its image plane. 

According to a particular aspect of the first embodiment, for each 
of the blocks (Bi), the data item of a second type ( w/™ }, indicative of a degree of 
significance of the visual content of the block (Bi) under consideration with 
respect to the overall content of the image (Im), is obtained by calculating the 
ratio between the Euclidean norm of the data item (A/™) of the first type 
associated with the block (B|) under consideration and the sum of the Euclidean 
norms of the data items of the first type associated with all the blocks of the 
image (Im). 

This mode of calculating the data item of the second type ( w/"* ) 
associated with each of the blocks (Bi) of the image combines speed and 
efficiency. 

According to a second embodiment of the invention, the image 
plane of the image (Im) is divided according to a quadtree decomposition 
process by means of which, at each phase of the decomposition, a block under 
consideration, referred to as the "parent block" (B^), is decomposed into four 
blocks, referred to as "child blocks" {Bf ), equal in size to a quarter the size of 
the parent block, and whose combination gives the parent block, the 
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decomposition beginning with the overall image plane of the image (Im) and 
finishing when the predefined number N of blocks is reached. Moreover, at 
each phase of the decomposition, there is calculated, for each of the child 
blocks (B^), a data item (wj) of a second type, indicative of a degree of 
5 significance of the visual content of the child block under consideration with 
respect to the overall visual content of the parent block (5^). The second 
information item (W(lm)) is then composed of the set of data items ( w^) of the 
second type stored according to a quadtree structure, each node of which is 
constituted by one of the data items of the second type. 

10 Unlike the embodiment in which the image plane is divided 

according to a predefined grid of blocks, in this embodiment, the quadtree 
decomposition process, in which a number N of blocks to be reached is 
predefined, makes it possible to adapt the definition of the division grid to the 
spatial distribution of the content of the image. 

15 According to a particular aspect of the second embodiment, at 

each phase of the quadtree decomposition of the image (Im), there is extracted, 
from the parent block {B^ ) under consideration, a data item {h^)o1 the first type 
representing at least one characteristic of its visual content and, for each child 
block (B^) obtained by decomposition of the parent block (B^), there is 

20 extracted a data item {h^) of the first type representing at least one 
characteristic of the visual content of the child block under consideration. The 
data item { ) of the second type indicative of a degree of significance of the 
visual content of a child block (B^) under consideration with respect to the 
overall visual content of the corresponding parent block (B^) Is obtained by 

25 calculating the ratio between the Euclidean norm of the data item (A/ ) of the 
first type extracted for the child block (B^) under consideration and the 
Euclidean norm of the data item (h^) of the first type extracted from the 
corresponding parent block {B^ ). 



For each newly created child block, there is thus calculated a data 
item ( w^) of the second type indicative of a degree of significance of the visual 
content of the child block with respect to the overall visual content of the 
corresponding parent block. Therefore, the block subdivision process can be 
iterative. 

According to a particular implementation characteristic of the 
invention, each of the data items of the first type, representing at least one 
characteristic of the visual content of a block under consideration of an image 
to be indexed, represents the distribution of colours in the block. 

The present invention concerns, according to a second aspect, a 
method of searching for images, from an example image, in a database in 
which digital images are stored. In accordance with the invention, this image 
search method is characterised in that the example image and each of the 
images stored in the database are indexed according to an image indexing 
method as defined above. 

With each of the images in the database being associated with an 
index as defined above, an image search according to such a method will 
consequently have an increased reliability/accuracy. Furthermore, with the 
indices being composed of two information items (H(lm) and W(lm)), a 
hierarchical type search method can be implemented. 

In accordance with a preferred embodiment of the invention, the 
image search method has the following steps: 

- calculating a first similarity (Filter) between the example image 
(Q) and each of the images (D) amongst a predefined plurality of stored 
images, this first similarity being calculated from the second information items 
(W(Q); W(D)) associated respectively with the example image (Q) and the 
stored image (D) under consideration; 

- providing a first subset of images selected from amongst the 
predefined plurality of images according to their degree of first similarity (Filter) 
with the example image (Q); 



- calculating a second similarity (Match) between the example 
image (Q) and each of the images (Ds) amongst the first subset of selected 
images, this second similarity being calculated from the first information items 
(H(Q); H(Ds) associated respectively with the example image (Q) and the 

5 selected image (Ds) under consideration; 

- providing at least one image referred to as a result image, this 
result image or these result images being selected from amongst the first 
subset of selected images, according to its degree of second similarity (Match) 
with the example image. 

10 Such a hierarchical search method makes it possible to obtain 

response times to a query which are small in comparison with those which 
would be obtained by using a linear search method based solely on the first 
information items (H(lm)). This difference is all the greater, the larger the 
number of images in the database. 

15 According to a first embodiment of the search method, when the 

images are indexed in accordance with the above-mentioned first embodiment 
of the indexing method in accordance with the present invention, the step of 
calculating a first similarity (Filter) between the example image (Q) and each of 
the images (D) amongst a predefined plurality of stored images is implemented 

20 by calculating a distance between the second information item (W(Q)) 
associated with the example image and the second information item (W(D)) 
associated with the stored image under consideration (D). 

The second information items (W(lm)) which are contained in the 
indices associated with the images in the database are by nature not very 
25 discriminant; however, a calculation of similarity, performed as described 
above, between two of these information items is very quick. Consequently, the 
calculation of a "first similarity" between the example image and each of the 
images in the database is quick overall. 

According to a second embodiment of the search method, when 
30 the images are indexed in accordance with the above-mentioned second 
embodiment of the indexing method in accordance with the present invention. 



the step of calculating an aforementioned first similarity (Filter) is implemented 
by an isomorphism detection method applied to the quadtrees representing the 
second information items (W(Q); W(D)) associated respectively with the 
example image (Q) and the stored image (D) under consideration. 

The use of an isomorphism detection method allows a quick 
comparison of the structure of two graphs. Knowing that the information 
concerning the spatial distribution of the content of an image is determined by a 
graph structure, two images are considered to be similar (first similarity) if the 
graph structures associated with their respective Information items are close to 
one another. 

According to a particular characteristic of the search method 
according to the invention, the step of calculating a second similarity (Match) 
between the example image (Q) and each of the images (Ds) amongst the first 
subset of selected images is implemented by calculating a distance between 
the first information item (H(Q)) associated with the example image and the first 
information item (H(D)) associated with the stored Image under consideration 
(D). 

This second similarity makes it possible to compare the content of 
two images according to the first information item (H(lm)) associated with each 
of them. The measurement of this similarity is expensive in calculation time but 
makes it possible to obtain accurate responses to a given search query. 

More particularly, the step of calculating the second similarity 
(Match) is implemented by calculating the sum of the distances between each 
of the components {hf) of the first information item (H(Q)) associated with the 
example image (Q) and the corresponding component {h^') of the first 
information item (H(Ds)) associated with the stored Image (Ds) under 
consideration. 

According to another aspect, the present invention concerns a 
digital data processing device, characterised in that it comprises means 
adapted to implement an image indexing method and/or an image search 
method as defined previously. 
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The present invention also concerns a connputer connprising such 
a digital data processing device. 

The invention also relates to a computer program having one or 
more instruction sequences capable of implementing the image indexing 
method and/or the image search method according to the invention when the 
program is loaded and executed in a computer. 

The invention also relates to an information medium, such as a 
diskette or a compact disk (CD), characterised in that it contains such a 
computer program. 

The advantages of this device, this computer, this computer 
program and this information medium are identical to those of the methods as 
briefly explained above. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Other features and advantages of the invention will also emerge in 
the description below, given with reference to the accompanying drawings in 
which: 

- Figure 1 is a block diagram illustrating the general architecture of 
an image search device in which the invention can be implemented; 

- Figure 2 depicts a flow diagram illustrating the sequence of steps 
of a hierarchical image search method implemented in the device of Figure 1; 

- Figure 3 depicts a flow diagram illustrating the general principle 
of an image Indexing method according to the invention; 

- Figure 4 illustrates an example of dividing an image according to 
a first embodiment of the invention; 

- Figure 5 illustrates an example of dividing an image according to 
a second embodiment of the invention; 

- Figure 6, composed of Figures 6a and 6b, illustrates an image 
search method according to the invention comprising a first, so-called "filtering" 
phase (Fig. 6a), and a second, so-called "matching" phase (Fig. 6b); 

- Figure 7 depicts schematically a computer adapted to implement 
the image indexing and/or search methods in accordance with the present 
invention. 



11 



DETAILED DESCRIPTION OF THE INVENTION 
With reference to Figure 1, there will now be described the 
general architecture of an image search device in which the present invention 
can be implemented. 

As depicted in Figure 1, this image search device has a database 
1 1 containing digital images stored in compressed or uncompressed form. 

An image data input/output unit 10 allows input into the database 
1 1 of new images to be stored and retrieval of stored images and/or the data 
indexed (indices) to stored images. 

An example image input unit 30 is associated with an input device 
31 in order to allow a user to input an example image to serve as a reference 
for the search. Typically, the input device 31 has a pointing device such as a 
mouse, and a keyboard. 

An image data storage unit 20 is intended to store temporarily the 
data retrieved from the database 1 1 or the data associated with the example 
image obtained by the units 30 and 31. 

A unit 40 has the function of retrieving the data indexed to the 
images stored in the database or generating these data for the new images 
which are to be stored in the database. 

Similarly, a unit 50 is responsible for generating or retrieving the 
index data associated with the example image. 

The index associated with each stored image is composed of two 
sub-indices, a first, not very discriminant sub-index and a second, more 
discriminant sub-index. 

A similarity calculation unit 60 is responsible for evaluating the 
similarity of the example image with the images in the database. 

An image selection unit 70 is responsible for sorting the images in 
the database according to their similarity with the example image and selecting 
one or more images as the result of the search. 

The search method being of the hierarchical type, calculation of 
the similarity is performed in two steps as mentioned above. A start is made 
with a filtering step, followed by a "matching" step. 
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A display unit 90, comprising typically a screen, nnakes it possible 
to display windows for dialogue with the user as well as the result of an image 
search, represented for example in the form of "image thumbnails". 

Finally, a control unit 80 controls and manages the overall 
operation of the image search device. 

It should be noted that the various units which have just been 
described are implemented by the assembly of hardware and software 
elements. 

There will now be described, in connection with Figure 2, a 
hierarchical image search method implemented in the device of Figure 1. 

The image search method starts with the step SI at which a 
human user uses the example image input unit 30 associated with the input 
device 31, in order to define an example image which will sen/e as a reference 
for the image search in the database. 

The user has the choice for defining the example image, between 
designating an image from the database or providing an image external to the 
database. 

In order to be able to choose an image from the database as the 
example image, the user has the capability of requesting the display unit 90 to 
display, on the screen, "thumbnails" corresponding to the images in the 
database. The user can then select, by means of, for example, a mouse, the 
thumbnail whose content seems to him to be characteristic of that of the image 
or images for which he is searching. In a variant, the user can also use the 
keyboard to input the reference of the stored image he is choosing as the 
example image. Finally, if the user chooses to provide an example image 
external to the database, he can for example provide, to the unit 30, the access 
path to this image, which will be accessible via a data acquisition peripheral 
such as a diskette or compact disk (CDROM) drive integrated with the input 
device 31 . 

Returning to Figure 2, once an example image has been input by 
the user (step SI), at the step S3, the example image index extraction unit 50 
retrieves or generates the index of the example image. 
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Similarly, at the following step S5, the stored image index 
extraction unit 40 extracts the index associated with each of the images stored 
in the database. 

The indices associated with the example image and with the 
stored images are temporarily saved in the image data storage unit 20. 

At the step S7, the similarity calcuiation unit 60 retrieves the 
indices of the stored images and the index of the example image, previously 
stored in the unit 20, and performs a calculation of a first similarity between the 
example image and each of the stored images whose index has been extracted 
from the database. This "first similarity" is calculated on the basis of the first 
sub-indices (not very discriminant) associated with the example image and with 
the stored images. 

At the step S9 which follows, the image selection unit 70 carries 
out the sorting and a selection of the stored images which were compared with 
the example image, according to their degree of first similarity with the said 
example image. For example, only the stored Images whose calculated degree 
of similarity is above a predefined threshold will be kept. Amongst these, only a 
predefined number of images (for example 50) will be retained as having the 
highest degree of first similarity with the example image. 

The steps S7 and S9 which have just been described constitute 
the "filtering" phase of the hierarchical search process. At the end of this phase, 
a subset of images stored in the database has therefore been selected. 

At the step S11 which follows, there is carried out the calculation 
of a second similarity performed on the basis of the second sub-indices (more 
discriminant) associated with the example image and with the stored images 
belonging to the subset of images obtained during the filtering phase. 

At the step SI 3 which follows, the image selection unit 70 carries 
out the sorting and a selection of the images of the subset of images which 
were compared with the example image, according to their degree of second 
similarity with the said example image. For example, only the stored images 
whose calculated degree of similarity is above a predefined threshold will be 
kept. Amongst these, only a predefined number of images (for example 5) will 
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be retained as having the highest degree of second similarity with the example 
image. 

The steps S1 1 and S13 which have just been described constitute 
the "matching" phase of the hierarchical search process. At the end of this 
phase, a predetermined number of images in the database have been selected 
as being the result of the search. 

This result is displayed by the display unit 90 on a screen. In 
practice, the display unit displays, on the screen, thumbnails associated with 
the selected images. On the screen, the thumbnails are typically arranged 
according to an increasing or decreasing order of similarity with the example 
image. The user can then choose one or more images he considers match his 
query. 

With reference to Figure 3, there will now be described the 
general principle of an image indexing method in conformity with the invention. 
According to a preferred embodiment of the invention, this method is 
implemented in an image search device as described previously in connection 
with Figure 1 . 

In accordance with the invention, this method of indexing a given 
image, denoted Im, essentially has three steps: a step S31 for generating a first 
information item, referred to as a "first sub-index" and denoted H(lm), 
characteristic of the visual content of the image; a step S32 for generating a 
second information item, referred to as a "second sub-index" and denoted 
W(lm), characteristic of the spatial distribution in the image plane of the visual 
content of the image; and finally a step S33 for associating, with the image Im, 
an index, denoted IDX(lm), and composed of the aforementioned two 
information items. 

The index IDX(lm) thus obtained is stored in the database 11 in 
relation with the storage representation of the image Im under consideration, 
that is to say, for example, with the corresponding binary stream in the case of 
storage in compressed form, or with a bitmap representation in the case of 
storage in uncompressed form. 
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As depicted in Figure 3, the step S31 for generating the 
information item characteristic of the visual content of the image is composed of 
the sub-steps S310 to S315. 

At the step S310, the image Im to be indexed is divided into an 
5 integer number N of blocks denoted Bi (with i between 1 and N) forming a 
partitioning of the image Im. 

It should be stated here that the term "partitioning" must be 
understood here In its mathematical sense which is stated below. 

A partitioning of a set is a division of that set into non-empty parts, 
10 disjoint in pairs, and whose combination is equal to the set. 

The process according to which the image is divided will be 
described later in connection with Figures 4 and 5. 

During the following two steps, a start is made by selecting (S311) 
a block Bi and then there is extracted (S312), from this block, a data item, 
1 5 denoted /z/"" , characteristic of the visual content of the block. 

At the following step S313, the extracted data item ^/'"is saved 
temporarily in the image data storage unit 20. 

The step S314 which follows is a test step in which it is 
determined whether or not all the blocks resulting from the division of the image 
20 Im have been processed. If not, a return is made to the step S311 in order to 
select another block and the process starts again. 

If yes, this means that, for each block Bj , a data item h!"" has been 
extracted and saved. In this case, the next step will be the step S315, in which 
there is generated the information item (H(lm)) characteristic of the visual 
25 content of the image Im (that is to say the first sub-index) as being the vector 
with N components (N: number of blocks), each of which is one of the data 
items hj'" calculated during the step S312. H(lm) can therefore be expressed 
as follows: 

/f(Im)-[/j^^™,/?f,...,C] 
30 According to a preferred embodiment of the invention, each of the 

data items hj"" characteristic of the visual content of a block under 
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consideration of the image to be indexed (Im) represents tlie distribution of 
colours in the block. 

in practice, each of the data items hj"" is a colour histogram 
calculated for the block Bi selected (S31 1). 

The method used within the context of the present invention for 
calculating a colour histogram from an image is known in the prior art. To obtain 
further details on image indexing based on colours, reference can be made for 
example to the article by M. J. Swain and D. H. Ballard, entitled "Color 
Indexing", International Journal of Computer Vision, 7:1, 1991. 

With reference to Figure 4, there will now be described an 
example of dividing an image according to a first embodiment of the invention. 

In the division example illustrated in Figure 4, the image Im (2) to 
be processed is divided according to a rectangular grid into 16 blocks (N=16), 
Bi to Bi6, with the same surface area. 

Still according to the first embodiment of the invention, there will 
now be detailed the step (cf. Fig. 3, S32) for generating a second information 
item, referred to as the "second sub-index" and denoted W(lm), characteristic 
of the spatial distribution in the image plane of the visual content of the image 
im to be indexed. 

In accordance with the invention, in this embodiment, the step for 
generating the second sub-index W{lm) has the following sub-steps. 

During a first sub-step, there is calculated, for each of the blocks 
(Bi) of the image, a data item, denoted w,"", indicating a degree of significance 
of the visual content of the block Bi under consideration with respect to the 
overall content of the image Im. 

During a second sub-step, there is generated the second sub- 
index W(lm) as being the vector with N components (N: number of blocks), 
each component of which is one of the data items wf" calculated during the 
aforementioned first sub-step. W(lm) can therefore be expressed as follows: 
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Thus, this second sub-index is effectively characteristic of the 
spatial distribution of the content of the image Im in its Image plane. 

In conformity with a preferred embodiment of the invention, for 
each of the blocks Bi of the Image to be indexed, the data item w-"" is obtained 
5 which is indicative of a degree of significance of the visual content of the block 
Bi under consideration with respect to the overall content of the image Im, by 
applying the following formula: 

(1) 

in which is the data Item characteristic of the visual content of 
10 the block under consideration, extracted during generation of the first sub-index 
H(lm) of the image (cf. Fig. 3, S312). As mentioned previously, in practice, each 
of the data items ^/^ is a colour histogram calculated for the block Bi under 
consideration. 

Thus, each of the data items w/" Is obtained by calculating the 
15 ratio between the Euclidean norm of the data item A/'" characteristic of the 
visual content of the block under consideration and the sum of the Euclidean 
norms of these data Items hi"" associated with all the blocks of the image Im. 

With reference to Figure 5, there will now be described an 
example of dividing an image according to a second embodiment of the 
20 invention. 

According to this second embodiment, the image plane of the 
image (Im) Is divided according to a quadtree decomposition process. 

At each phase of the decomposition, a block under consideration, 
referred to as a "parent block" and denoted , Is decomposed into four 
25 blocks, referred to as "child blocks" and denoted , equal in size to a quarter 
the size of the parent block, and whose combination gives the parent block. 

The decomposition starts with the overall image plane of the 
image Im and finishes when a predefined number N of blocks is reached. 



For example, if the predefined number of blocks is 16, the 
decomposition will be performed in two phases only. 

As depicted in Figure 5, during a phase under consideration, a 
parent block is decomposed into four child blocks: , Bf , B^ and B^ . 
5 In accordance with the invention, at each phase of the quadtree 

decomposition of the image, there is calculated, for each of the child blocks 
, a data item indicative of the degree of significance of the visual content 
of the child block under consideration with respect to the overall visual content 
of the corresponding parent block . 

10 The set of these data items calculated during each of the 

phases of the decomposition is then stored according to a quadtree structure, 
each node of which is constituted by one of these data items. 

There is then generated the second sub-index W(lm) as being 
composed of the set of data items stored according to the quadtree 

15 structure. 

There will now be detailed, in conformity with a preferred 
embodiment of the invention, the method according to which there is calculated 
a data item indicative of the degree of significance of the visual content of a 
child block with respect to the overall visual content of the corresponding parent 
20 block B^ . 

At each phase of the quadtree decomposition of the image Im to 
be indexed, there is extracted, from the parent block B^ under consideration, a 
data item, denoted h^, representing at least one characteristic of its visual 
content and, for each child block B^ obtained by decomposition of the parent 
25 block Bp under consideration, there is extracted a data item, denoted , 
representing at least one characteristic of the visual content of the child block 
under consideration. 
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There is then obtained the data item indicative of the degree 
of significance of the visual content of a child block under consideration with 
respect to the overall visual content of the corresponding parent block , by 
applying the following formula: 



5 




(2) 



Thus, a data item calculated for a child block under 
consideration B^ is obtained by calculating the ratio between the Euclidean 
norm of the data item extracted from this child block B^ and the Euclidean 
norm of the data item extracted from the corresponding parent block B^ . 

10 According to a preferred embodiment of the invention, each of the 

data items , h^, , characteristic respectively of the visual content of a child 
block under consideration B^ of the image Im and of the visual content of the 
parent block ^^from which this child block originated, represents the 
distribution of colours in the block. 

15 in practice, each of the data items , is a colour histogram 

extracted from the corresponding block. 

Still in this embodiment of the invention, in which an image to be 
indexed Im is decomposed according to a quadtree, the first sub-index H(lm), 
that is to say the information item characteristic of the visual content of the 

20 image (Im), is constituted by the vector composed of the data items 

obtained during the last phase of decomposition of the image, that is to say, the 
data items extracted for the N child blocks obtained during this last phase. 

With reference to Figure 6, composed of Figures 6a and 6b, there 
will now be described a hierarchical search method based on an indexing 

25 technique in conformity with the present invention. 
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There will first be described the first so-called "filtering" phase in 
connection with Figure 6a, and there will next be described, in connection with 
Figure 6b, the so-called "matching" phase. 

In Figure 6a, the search method according to the present 
5 invention starts at the step S601 in which the index of the example image Q, 
denoted IDX(Q), is retrieved. The index of the example image, like the index of 
each of the images stored in the database, is composed of the first sub-index 
H(Q) characteristic of the visual content of the image, and the second sub- 
index W(Q) characteristic of the spatial distribution of the visual content of the 
10 image in its Image plane. 

At the following step, a counter i is initialized to zero. This counter 
will be used to count the number of images processed in the database. 

At the step S605 which follows, an image, denoted D, is selected 
from the database, for the purpose of being compared with the example image. 
15 To that end, there is retrieved, at the step S607, the index, denoted IDX(D), 
associated with this image. 

It should be noted here that, although in a preferred embodiment 
of the invention all the images stored in the database are compared with the 
example image, also, as a variant, a number or a subset of stored images with 
20 which the search will be concerned can be predefined. In the following, the 
number of stored images with which the search is concerned is denoted "NB". 

At the step S609, there is calculated a first similarity, denoted 
Filter{Q,D) , between the example image Q and the image under consideration 
D. This first similarity is calculated from the sub-indices W(Q) and W(D) 
25 associated respectively with the example image and with the stored image 
under consideration. 

In the embodiment of the invention described in relation to Figure 

4, this gives: 

W(Q)=[w^,wf,...,w^] and = [w/',wf ,...,<] 

30 where each of the data items w9 and wf is indicative of a degree 

of significance of the visual content of the block under consideration of the 



example image or of the stored image under consideration with respect to the 
overall content of the image (Q or D). 

In this embodiment, the calculation of the first similarity 
Filter(Q,D) is implemented by determining the Euclidean distance between the 
5 sub-index W(Q) associated with the example image, and the corresponding 
sub-index W(D) associated with the stored image under consideration. 

In the embodiment of the invention described in relation to Figure 
5, in which any image to be indexed is decomposed according to a quadtree 
structure, each sub-index, W(Q), W(D), is composed of the set of data items 
10 denoted and stored according to the quadtree structure (cf. above, 
description in connection with Figure 5). 

In this embodiment, the calculation of the first similarity, 
Filter{Q,D) , is implemented by an isomorphism detection method applied to 
the quadtrees representing the sub-indices W(Q) and W(D) associated 
15 respectively with the example image Q and the stored image D under 
consideration. 

To obtain further information concerning graph isomorphism 
detection methods, reference can be made for example to the work by B. T. 
Messmer, entitled "Subgraph isomorphism in polynomial time", thesis, 
20 University of Bern, November 1995. 

Returning to Figure 6a, once the first similarity Filter{Q,D) has 
been calculated (S609), the result of this calculation is saved temporarily in 
memory (unit 20, Fig. 1 ) at the step S61 1 . 

The following step S613 is a test step in which the counter 1 is first 
25 incremented and then its value is compared with the number NB of stored 
images with which the search is concerned. 

If the value of the counter i is strictly greater than the number NB, 
this means that a first similarity Filter{Q,D) has been calculated for all the NB 
stored images. In this case, the next step will be the following step S615. 
30 Conversely, if the value of the counter i is less than or equal to the 

number NB, this means that not all the stored images under consideration for 
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the search have been processed. In this case, the next step will be the step 
E605 again in order to select another stored image D, and the process starts 
again as described previously {S605 to S61 1). 

At the step S615, all the similarity values Filter{Q,D) stored for 
5 the NB stored images are retrieved, and then these values are sorted. A 
selection is then made, according to the similarity values obtained, of a 
predefined number, denoted "NB1", (for example equal to 50), of images 
amongst the NB stored images, as being the most similar to the example 
image. 

10 Finally, at the step S617, the indices of the NB1 selected images 

are saved. 

Thus, at the end of the filtering phase of the search method, there 
has been selected, from amongst the images in the database, a subset 
composed of a number NB1 of stored images. In the following description, any 
15 image amongst the NB1 stored images resulting from the filtering is designated 
by "image Ds". 

Once the filtering phase has been carried out, the next phase is 
the "matching" phase of the search process which starts with the step E619 of 
Figure 6b. 

20 The step S619 is an initialization step in which the counter i is 

reset to zero. The counter i will be used to count the images Ds processed 
during this phase. 

At the following step S621, an image Ds is selected (more 
precisely, its index is retrieved). 

25 At the step S623, there is calculated a second similarity, denoted 

MatchiQ,Ds) , between the example image Q and the stored image Ds under 
consideration. This second similarity is calculated from the sub-indices H(Q) 
and H(Ds) associated respectively with the example image and the stored 
image under consideration, and expresses a distance between the two sub- 

30 indices H(Q) and H(Ds). 

As described previously in connection with Figure 3 (step S31), 
each of the sub-indices H(Q) and H(Ds) is a vector with N components where N 
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is the number of blocks resulting from the division of the images. Each of these 
components is Gharacteristic of the visual content of a corresponding block. 
This gives: 

H{Q)^[hf,h^,...,h°] and H{Ds)^[h^' X\...X'] 
5 According to a preferred embodiment of the invention, this second 

similarity Match(Q,Ds) is obtained by calculating the sum of the distances 
between each of the components of the vector H(Q) associated with the 
example image and the corresponding component of the vector H(Ds) 
associated with the stored image Ds under consideration. 
1 0 This can be expressed by th^ following formula: 

MatchiQ, Ds) = ^ d{hf , hf' ) (3) 

According to a preferred embodiment in which the components 
hf,hf'aTe colour histograms associated with the corresponding blocks of the 
images Q and Ds, each of the distances d{hf,h^')\s obtained by calculating 
1 5 the intersection between these histograms {hfand hf^' ). 

It should be noted that the intersection between histograms is 
known in the prior art. To obtain further details on the intersection between 
histograms, reference can be made, for example, to the article by M. J. Swain 
and D. H. Ballard, entitled "Color Indexing", International Journal of Computer 
20 Vision, 7:1, 1991. 

Returning to Figure 6b, once the calculation of the second 
similarity Match(Q,Ds) has been performed (S623), the result of this calculation 
is saved temporarily in memory (unit 20, Fig. 1) at the step S625. 

The following step S627 is a test step in which the counter i is first 
25 incremented and then its value is compared with the number NB1 of stored 
images selected during the filtering phase (Fig. 6a). 

If the value of the counter i is strictly greater than the number 
NB1, this means that a second similarity Match{Q,Ds) has been calculated for 
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all the NB1 stored images selected. In this case, the next step is the following 
step S629. 

Conversely, if the value of the counter i is less than or equal to the 
number NB1, this means that not all the stored images selected during the 
5 filtering phase have been processed. In this case, the next step is the step 
E621 again in order to select another stored image Ds, and the processes 
starts again as described previously (S621 to S627). 

At the step S629, all the similarity values Match{Q,Ds) stored for 
the NB1 stored images are retrieved, and then these values are sorted. 
10 Next a selection is made, according to the similarity values 

obtained, of a predefined number, denoted "NB2", (for example equal to 10) of 
images amongst the NB1 stored images selected during the filtering phase, as 
being the most similar to the example image. 

Finally, at the step S631, the indices of the NB2 images selected 
1 5 at the end of the matching phase are saved. 

These NB2 images constitute the final result of the search. There 
can then be displayed, on the screen, thumbnails corresponding to these 
images, so that the user can choose the one he considers matches his query 
best. 

20 With reference now to Figure 7, there will be described a 

computer adapted to implement the image indexing and/or search methods in 
accordance with the present invention and described above in connection with 
Figures 2 to 6. 

It should be noted that the computer illustrated in Figure 7 
25 constitutes a particular embodiment of the general device described above in 
relation to Figure 1 . 

In this embodiment, the image indexing and/or search methods 
according to the invention are implemented by a computer program. This 
program has one or more instruction sequences whose execution by the said 
30 computer allows the implementation of the steps of these methods. 

In Figure 7, the computer 3, which can be typically a micro- 
computer or a workstation, is connected to various peripherals, for example a 



digital camera 310 or any other image acquisition or storage device, such as a 
scanner, providing information (images, video) to the computer 3. These 
images can be stored in the storage means which the computer has, such as a 
hard disk 304. 

5 The computer 3 also has a communication interface 308 

connected to a communication network 309, for example the well-known 
Internet network, capable of transmitting digital information to the computer. 

The computer 3 also has data storage means such as a hard disk 
304, and a diskette drive 305 allowing writing of data on a diskette 305 and 
10 reading of these data. The computer may also have a compact disk (CDROM) 
drive (not depicted) on which there can be stored images, thus constituting a 
database, and a PC CARD reader (not depicted). 

According to a preferred embodiment of the invention, the 
executable code of the program making it possible to implement the image 
1 5 indexing and/or search methods is stored on the hard disk 304. 

According to a variant embodiment, the executable code of this 
program is stored in a read only memory (ROM) 300 of the computer. 

According to a second variant embodiment, the executable code 
of the program can be downloaded from the communication network 309 via 
20 the communication interface 308 in order to be stored on the hard disk 304. 

The computer 3 also has a screen 302 making it possible to view 
images and to be used as a graphical interface between the program and the 
user, the latter being able to formulate queries by means of, for example, a 
pointing device (not depicted) such as a mouse or light-pen, and a keyboard 
25 303. 

The computer has a central processing unit (CPU) 301, for 
example a microprocessor, which controls and directs the execution of the 
instructions of the program according to the invention stored in the read only 
memory 300 or on the hard disk 304. The central processing unit 301 then 
30 carries out the function of the control unit 80 described previously in relation to 
Figure 1. 
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The computer also has a random access memory (RAM) 307 
having registers intended to store the variables created and modified during 
execution of the program, notably the variables mentioned previously in relation 
to the description of Figures 2 and 6, as can be seen in the enlarged 
5 representation of the RAM in Figure 7. 

Finally, the computer has a communication bus 311 to allow 
communication and interoperability between the various aforementioned units 
making up the computer 3. 

Of course, many modifications can be made to the embodiments 
10 of the invention described above without departing from the scope of the 
invention. 



